Information processing apparatus, control method, and program

ABSTRACT

An information processing apparatus (2000) includes a product recognition unit (2020), a shelf label recognition unit (2040), and a determination unit (2060). The product recognition unit (2020) recognizes the number of types of products or positions of each type of products from a target image. The shelf label recognition unit (2040) recognizes the number or positions of shelf labels indicating information of the products from the target image. The determination unit (2060) determines whether or not there is an error in a relationship between the recognized number of types of products and the recognized number of shelf labels or in a relationship between the recognized positions of each type of products and the recognized positions of the shelf labels.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 17/675,113 filed on Feb. 18, 2022, which is acontinuation application of U.S. patent application Ser. No. 16/744,947filed on Jan. 16, 2020, which issued as U.S. Pat. No. 11,288,627, whichis a continuation application of U.S. patent application Ser. No.15/506,299 filed on Feb. 24, 2017, which issued as U.S. Pat. No.10,579,962, which is a National Stage Entry of international applicationPCT/JP2015/077268, filed on Sep. 28, 2015, which claims the benefit ofpriority from Japanese Patent Application 2014-201363 filed on Sep. 30,2014, the disclosures of all of which are incorporated in their entiretyby reference herein.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, acontrol method, and a program.

BACKGROUND ART

On a product shelving in a store, there is a shelf label near a productthat describes information such as a name or price of the product. Inrecent years, a system for supporting management of such the shelf labelis developed.

Patent Document 1 discloses a system in which store shelves have storelabels that display product information and shelf label identificationinformation, and the system performs: imaging shelf label identificationinformation of each shelf label with a camera, recognizing the imagedshelf label, and managing the shelf label. Specifically, this systemgenerates a map representing a position of each shelf label, or detectsthe shelf label the expiration date for use of which has been expiredand reports the detected shelf label.

RELATED DOCUMENT Patent Document

-   -   [Patent Document 1] Japanese Patent Application Publication No.        2014-48752

SUMMARY OF THE INVENTION Technical Problem

It is necessary to re-arrange, add or reduce the shelf labels accordingto a change of a line-up of products in a store. In addition, theposition of the shelf label is sometimes shifted erroneously by acustomer or a salesperson, and the position of the shelf label has to becorrected in such a case. If the customer or salesperson forgets tochange the arrangement of shelf labels or makes a mistake in changing ofthe arrangement of shelf labels, the number or positional relationshipbetween the shelf label and the product becomes inconsistent, and itcauses a problem that the customer is not able to appropriatelyrecognize the information of the product. The system of Patent Document1 does not disclose a technology for managing the consistency on thenumber or positional relationship between the shelf label and theproduct.

The invention has been made in view of the aforementioned problems. Thepresent invention provides a technology for supporting the checking ofarrangement of products or shelf labels.

Solution to Problem

An information processing apparatus provided by the present inventionincludes: a product recognition unit recognizing a number of types ofproducts or positions of each type of products from an image on whichthe products and shelf labels are imaged; a shelf label recognition unitrecognizing a number or positions of shelf labels indicating informationof each product from the image; and a determination unit determiningwhether or not there is an error in a relationship between therecognized number of types of products and the recognized number ofshelf labels or in a relationship between the recognized positions ofeach type of products and the recognized positions of the shelf labels.

A control method provided by the present invention is performed by acomputer. The control method includes: a product recognition step ofrecognizing a number of types of products or positions of each type ofproducts from an image on which the products and shelf labels areimaged; a shelf label recognition step of recognizing a number orpositions of shelf labels indicating information of the products fromthe image; and a determination step of determining whether or not thereis an error in a relationship between the recognized number of types ofproducts and the recognized number of shelf labels or in a relationshipbetween the recognized positions of each type of products and therecognized positions of the shelf labels.

A program provided by the present invention causes a computer to have afunction of operating as the information processing apparatus providedby the present invention.

Advantageous Effects of Invention

According to the present invention, a technology for supporting thechecking of arrangement of products or shelf labels is provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The above objects, other objects, advantages and features will be moreapparent from the following description of certain preferred embodimentstaken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram illustrating an information processingapparatus according to Exemplary Embodiment 1.

FIG. 2 is a flowchart illustrating a flow of processes performed by theinformation processing apparatus of Exemplary Embodiment 1.

FIG. 3 is a block diagram illustrating a hardware configuration of acomputer that realizes the information processing apparatus.

FIG. 4 is a diagram illustrating a target image.

FIG. 5 is a first diagram illustrating a target image representing acase where the number relationship is incorrect.

FIG. 6 is a second diagram illustrating the target image representing acase where the number relationship is incorrect.

FIG. 7 is a diagram illustrating a target image representing a casewhere the positional relationship is incorrect.

FIG. 8 is a diagram illustrating a target image representing a casewhere a shelf label is positioned in a boundary between two productgroups.

FIG. 9 is a diagram illustrating a zoomed area near the boundary betweenthe product groups of FIG. 8 .

FIG. 10 is a flowchart illustrating a flow of processes performed by theinformation processing apparatus of Exemplary Embodiment 1 in which boththe number relationship and the positional relationship are used.

FIG. 11 is a block diagram illustrating an information processingapparatus according to Exemplary Embodiment 2.

FIG. 12 is a flowchart illustrating a flow of processes performed by theinformation processing apparatus of Exemplary Embodiment 2.

DESCRIPTION OF EMBODIMENTS

Hereinafter, exemplary embodiments of the present invention will bedescribed with reference to the drawings. In all the drawings, the samecomponents will be assigned the same reference numerals, and thedescription thereof will not be appropriately repeated.

Exemplary Embodiment 1

FIG. 1 is a block diagram illustrating an information processingapparatus 2000 according to Exemplary Embodiment 1. In FIG. 1 , arrowsrepresent information flows. In FIG. 1 , each block depicts not aconfiguration of a hardware unit but a configuration of a functionalunit.

The information processing apparatus 2000 includes a product recognitionunit 2020, a shelf label recognition unit 2040, and a determination unit2060. The product recognition unit 2020 recognizes the number of typesof products or positions of each type of products from an image(hereinafter, referred to as a target image) on which products and shelflabels are imaged.

Here, “recognizing the number of types of products” means recognizing agroup of products having the same type as one type when that productshaving the same type are placed with being adjacent to one another. Forexample, suppose that three columns of Products A, two columns ofProducts B, and two columns of Products C are sequentially placed. Inthis case, the number of types of products is three. Hereinafter, thegroup of products having the same type arranged so as to be adjacent toone another as stated above is described as a product group. Note that,the number of products included in the product group may be one.

The “positions of each type of products” means positions of the productgroups. More specifically, in a case where three columns of Products A,two columns of Products B, and two columns of Products C aresequentially placed, the position of the product group of Products A,the position of the product group of Products B, and the position of theproduct group of Product C are the “positions of each type of products”.Concrete methods of how to represent the “position of the product group”or the like will be described below.

The shelf label recognition unit 2040 recognizes from the target imagethe number or positions of the shelf labels that indicate information ofeach product.

The determination unit 2060 determines an error in the relationshipbetween the number of the recognized types of products (the number ofproduct groups) and the number of shelf labels or the relationshipbetween the positions of each type of products (the positions of theproduct groups) and the shelf label.

<Flow of Processes>

FIG. 2 is a flowchart illustrating a flow of processes performed by theinformation processing apparatus 2000 of Exemplary Embodiment 1. Theinformation processing apparatus 2000 acquires the target image (S102).The product recognition unit 2020 recognizes the number or positions ofproduct groups from the target image (S104). The shelf label recognitionunit 2040 recognizes the number or positions of shelf labels from thetarget image (S106). The determination unit 2060 determines an error inthe relationship between the number of the recognized types of productgroups and the number of shelf labels or the relationship between theposition of the recognized product group and the position of the shelflabel (S108).

According to the information processing apparatus 2000 of the presentexemplary embodiment, an error is determined in the relationship betweenthe number of product groups (the number of the types of products) andthe number of shelf labels or in the relationship between the positionsof the product groups (the positions of each type of products) and thepositions of the shelf labels, by using the target image on which theproducts and the shelf labels are imaged. Thus, in a case where there isan error in the relationship between the number of product groups andthe number of shelf labels or in the relationship between the positionof the product group and the position of the shelf label, thesalesperson can appropriately set the shelf label and the product groupby correcting the number or position of the shelf labels or the productgroups. Note that, concrete situations in which there is an error in therelationship between the number of product groups and the number ofshelf labels or concrete situations in which there is an error in therelationship between the position of the product group and the positionof the shelf label will be described below in conjunction with concreteexamples.

Hereinafter, the information processing apparatus 2000 will be describedin more detail.

<Hardware Configuration Example>

The respective functional components of the information processingapparatus 2000 may be realized by hardware that realizes each functionalcomponent (for example, a hard-wired electronic circuit), or may berealized by a combination of hardware and software (for example, acombination of an electronic circuit and a program that controls theelectronic circuit). Hereinafter, in a case where each functionalcomponent is realized by the combination of hardware and software, theconfiguration thereof will be illustrated in detail.

The information processing apparatus 2000 may be realized by varioustypes of computers, such as a portable terminal, a personal computer(PC), or a server. Here, the information processing apparatus 2000 maybe realized by a dedicated computer for implementing the informationprocessing apparatus 2000, or may be realized by a general-purposecomputer that operates other applications.

FIG. 3 is a block diagram illustrating a hardware configuration of acomputer 5000 that realizes the information processing apparatus 2000.The computer 5000 includes a bus 5020, a processor 5040, a memory 5060,a storage 5080, and an input and output interface 5100. The bus 5020 isa data transmission path through which the processor 5040, the memory5060, the storage 5080, and the input and output interface 5100 transmitand receive data to and from one another. Here, the method of connectingthe processor 5040 and the like is not limited to bus connection. Theprocessor 5040 is, for example, a processing device such as a centralprocessing unit (CPU) or a graphics processing unit (GPU). The memory5060 is, for example, a memory such as a random access memory (RAM) or aread only memory (ROM). The storage 5080 is, for example, a storageapparatus such as a hard disk, a solid state drive (SSD), or a memorycard. The storage 5080 may be a memory such as RAM or ROM.

The input and output interface 5100 is an input and output interfacethrough which the computer 5000 transmits and receives data to and froman external apparatus. For example, in a case where the informationprocessing apparatus 2000 acquires the target image from the externalapparatus, the computer 5000 that realizes the information processingapparatus 2000 is connected to the external apparatus through the inputand output interface 5100. Note that, there are various methods ofconnecting the computer 5000 to the external apparatus through the inputand output interface. For example, such connection is bus connectionusing a bus line (for example, universal serial bus (USB)) or networkconnection using a network line. Note that the network line may be awireless line, or may be a wired line.

The storage 5080 stores a program for realizing the function of theinformation processing apparatus 2000. Specifically, program modules forrespectively realizing the functions of the product recognition unit2020, the shelf label recognition unit 2040, and the determination unit2060 are stored. The processor 5040 realizes the functions of theproduct recognition unit 2020, the shelf label recognition unit 2040,and the determination unit 2060 by executing the respective programmodules. Here, when the respective modules are executed, the processor5040 may read these modules into the memory 5060 and may execute theread module, or may execute the module without reading the module intothe memory 5060.

Note that, the hardware configuration of the computer 5000 used forrealizing the information processing apparatus 2000 is not limited tothe configuration illustrated in FIG. 3 . For example, the respectiveprogram module may be stored in the memory 5060. In this case, thecomputer 5000 may not include the storage 5080.

<Acquisition Method of Target Image>

The target image is imaged by a camera provided in a store. Only onetype of product or one shelf label may be pictured or a plurality ofdifferent products or a plurality of shelf labels may be pictured on onetarget image.

For example, the information processing apparatus 2000 acquires thetarget image stored within this camera. In a case where an externalstorage device is stored in the target image imaged by the camera, theinformation processing apparatus 2000 may acquire the target image fromthis storage device. The camera may be provided such that the imagedtarget image is stored in a storage device provided within theinformation processing apparatus 2000. In this case, the informationprocessing apparatus 2000 acquires the target image from a storage unitprovided within the information processing apparatus 2000.

The camera may be provided integrally with the computer that realizesthe information processing apparatus 2000, or may be provided outsidethe computer. As an example in which the camera is provided integrallywith the computer that realizes the information processing apparatus2000, the information processing apparatus 2000 may be realized by aportable terminal equipped with the camera. In this case, theinformation processing apparatus 2000 performs a process on the targetimage imaged by the camera of the portable terminal.

<Details of Product Recognition Unit 2020>

The product recognition unit 2020 analyzes the target image, andrecognizes a product. Here, a technique for recognizing an object suchas a product pictured on an image is a well-known technique, and thusvarious well-known techniques may be used for recognizing the object.Hereinafter, an example of a process performed by the productrecognition unit 2020 will be described.

FIG. 4 is a diagram illustrating the target image 10. A product shelvingon which the products are exhibited is pictured on the target image 10.Note that, the way of exhibiting the products is not limited to exhibitthe products on the product shelving, as long as the products and theshelf labels may be pictured on the target image.

An outer frame 20 is an outer frame of the product shelving. A shelfplate 30 is a plate on which the products are placed. A product group 40represents a product group. Three product groups are pictured on thetarget image 10 of FIG. 4 . A shelf label 50 is a label displayinginformation of the product. In FIG. 4 , three shelf labels are picturedon the target image 10. Here, the shelf label 50 may be a shelf labelcreated as an analog medium such as paper, or may be a shelf labelcreated as an electronic medium.

Initially, the product recognition unit 2020 recognizes the product fromthe target image 10. The product recognition unit 2020 recognizes agroup of products among the recognized products that are adjacent toeach other and have similar feature values, as one product group 40. Forexample, the product group 40-1, the product group 40-2, and the productgroup 40-3 are recognized from the target image 10 in FIG. 4 .

<<Method 1 of Recognizing Product Group 40>>

For example, the product recognition unit 2020 classifies the respectiveproducts into the product groups by sequentially comparing the featurevalues of the adjacent products from any of the left and the right.Specifically, the product recognition unit 2020 classifies theseproducts into the same product group 40 if the feature values of theadjacent products are the same as or similar to each other. On the otherhand, the product recognition unit 2020 classifies these products intodifferent product groups 40 if the feature values of the adjacentproducts are not similar to each other.

For example, in the example of FIG. 4 , the product recognition unit2020 sequentially compares the feature values of the respective productsplaced on a shelf plate from the left. Initially, the productrecognition unit 2020 determines that the feature values of the firstproduct and the second product are similar to each other, and classifiesthe first product and the second product into the same product group40-1. Similarly, the product recognition unit 2020 determines that thefeature values of the second product and the third product are similarto each other, and also classifies the third product into the sameproduct group 40-1. On the other hand, as a result acquired by comparingthe feature values of the third product and the fourth product, theproduct recognition unit 2020 determines that these feature values arenot similar. As a result, the fourth product is classified into theproduct group 40-2, which is a product group different from the productgroup 40-1.

According to this method, the product recognition unit 2020 does notneed to recognize the feature values of the respective products inadvance. Thus, it is possible to reduce computer resources required torealize the information processing apparatus 2000.

<<Method 2 of Recognizing Product Group 40>>

The product recognition unit 2020 may recognize the product group 40 byusing information (hereinafter, referred to as product information) thatis acquired by associating the product with a feature value describing afeature of the appearance of the product. The product information may bestored in the storage unit provided inside or outside the informationprocessing apparatus 2000 in advance. The product recognition unit 2020acquires the product information from the storage unit.

As for the product information, one or a plurality of feature values maybe associated with one product. Here, in a case where the plurality offeatures is associated with one product, these feature values are, forexample, feature values indicating the features of the product viewed indifferent directions. In some products, the appearance of the productlooks very different depending on directions in which the product isviewed. For example, a certain product would have patterns on the labelattached thereto that look very different between when being viewed in acertain direction and when being viewed in another direction. Since theexhibited products are not necessarily placed to face to the samedirection, when the appearance of the product looks very differentdepending on the directions in which the product is viewed, the featurevalues of the product extracted from the target image may be differenteven though the same product is viewed.

Thus, the product recognition unit 2020 determines the productcorresponding to the calculated feature values, by comparing the featurevalues calculated from the target image 10 with the feature valueindicated by the product information. The product recognition unit 2020classifies the products that are the same product as each other and areadjacent to each other into the same product group 40.

According to this method, the same type of products (products placed indifferent directions), the feature values calculated from the targetimage of which are not similar to each other, are classified into thesame product group 40. Thus, it is possible to increase the accuracy ofthe classification of the product group 40.

<<Method of Recognizing Number of Product Group 40>>

The product recognition unit 2020 recognizes the number of the productgroup 40 by counting the number of recognized product groups 40.

<<Method of Recognizing Position of Product Group 40>>

For example, the product recognition unit 2020 recognizes a coordinateof each product group 40 on the target image 10, as the position of theproduct group 40. Here, the coordinate of the product group 40 is acoordinate satisfying a predetermined condition within an areaindicating the product group 40. For example, this condition is acenter, a left end, or a right end of the area indicating the productgroup 40. Note that, the product recognition unit 2020 may represent theposition of the product group 40 as a two-dimensional coordinate (acombination of a horizontal direction with a vertical direction), or mayrepresent the position thereof as a one-dimensional coordinate (acoordinate in the horizontal direction).

<Details of Shelf Label Recognition Unit 2040>

The shelf label recognition unit 2040 analyzes a target image, andrecognizes a shelf label. Here, similarly to the product recognitionunit 2020, the shelf label recognition unit 2040 can use variousexisting techniques for recognizing an object from the image.Hereinafter, an example of a process performed by the shelf labelrecognition unit 2040 will be described.

<<Recognition Method of Shelf Label 50>>

Similarly to the case of the product recognition unit 2020, therecognition method of the shelf label will be described with referenceto FIG. 4 . For example, the shelf label recognition unit 2040recognizes the shelf label 50 by acquiring information indicating afeature value of the shelf label 50 and recognizing an object matchingthis feature value from the target image. Here, the feature value of theshelf label 50 may be stored in the storage unit provided inside oroutside the information processing apparatus 2000 in advance.

For example, the shelf label recognition unit 2040 may recognize thearea indicating the shelf plate 30, and then recognize as the shelflabel 50 an object having a predetermined shape (a rectangle or thelike) included in this area indicating the shelf plate 30. For example,the shelf label recognition unit 2040 recognizes the shelf plate byacquiring the feature value indicating the shelf plate and using theacquired feature value. Here, the feature value of the shelf plate 30 orthe predetermined shape indicating the shelf label 50 may be stored inthe storage unit provided inside or outside the information processingapparatus 2000 in advance.

<<Method of Recognizing Number of Shelf Label>

The shelf label recognition unit 2040 recognizes the number of shelflabels 50 by counting the number of shelf labels 50 recognized by theaforementioned method.

<<Method of Recognizing Position of Shelf Label>

The method of causing the shelf label recognition unit 2040 to recognizethe position of the shelf label 50 is the same as the method of causingthe product recognition unit 2020 to recognize the position of theproduct group 40.

<Details of Determination Unit 2060>

As stated above, the determination unit 2060 determines an error in therelationship between the number of recognized product groups 40 and thenumber of shelf labels 50, or the relationship between the position ofthe product group 40 and the position of the shelf label 50.

<<Determination Method of Number Relationship>>

The determination unit 2060 compares the number of product groups 40with the number of shelf labels 50. For example, the determination unit2060 determines that there is an error in the relationship between thenumbers if the number of product groups 40 and the number of shelflabels are different.

The determination method will be described in detail with reference toFIGS. 4 to 6 . In the aforementioned example of FIG. 4 , the number ofproduct groups 40 is 3, and the number of shelf labels 50 is 3. Thus,the determination unit 2060 determines that the relationship between thenumber of product groups 40 and the number of the shelf labels 50 iscorrect.

FIG. 5 is a first diagram illustrating the target image representing acase where the number relationship is incorrect. In the aforementionedexample of FIG. 5 , the number of product groups 40 is 3, whereas thenumber of shelf labels 50 is 2. Thus, the determination unit 2060determines that there is an error in the relationship between the numberof product groups 40 and the number of shelf labels 50. For example, thesituation illustrated by FIG. 5 may occur when the salesperson forgetsan operation of adding a shelf label 50-3 even though the product in theproduct group 40-3 is newly exhibited or when the shelf label 50-3 fallsoff. The salesperson can recognize that such a situation occurs, byusing the result of the determination performed by the determinationunit 2060.

FIG. 6 is a second diagram illustrating the target image representing acase where the number relationship is incorrect. In the example of FIG.6 , the number of product groups 40 is 2, whereas the number of shelflabels 50 is 3. In this case, the determination unit 2060 determinesthat there is an error in the relationship between the number of productgroups 40 and the number of shelf labels 50. For example, the situationillustrated by FIG. 6 may occur when the salesperson forgets removingthe shelf label 50-3 even though the exhibition of the product of theproduct group is terminated, when the salesperson forgets adding theproduct even though the shelf label 50-3 is added in order to newlyexhibit the product of the product group 40-3, or the product of theproduct group 40-3 is sold out. The determination result performed bythe determination unit 2060 is used, and thus, the salesperson canrecognize that such a status occurs.

<<Determination Method of Positional Relationship>>

For example, if there is not the shelf label 50 in a predetermineddirection to the product group 40, the determination unit 2060determines that there is an error in the relationship between theposition of the product group 40 and the position of the shelf label 50.For example, the predetermined direction is up, down, left, or right.

Here, as described above, suppose that information indicating adirection in which the shelf label 50 is attached for the product group40 is previously stored in the storage unit inside or outside theinformation processing apparatus 2000. For example, in a case where theshelf label 50 is attached above the product group 40, if there is noshelf label 50 above the product group 40, the determination unit 2060determines that there is an error in the relationship between theposition of the product group 40 and the position of the shelf label 50.

The process performed by the determination unit 2060 will be describedin detail with reference to FIGS. 4 and 7 . In these diagrams, it isassumed that the shelf label 50 is attached under the product group 40.

In FIG. 4 , a shelf label 50-1 is located under the product group 40-1,a shelf label 50-2 is located under the product group 40-2, and a shelflabel 50-3 is located under the product group 40-3. Thus, thedetermination unit 2060 determines that the relationship between theposition of product groups 40 and the position of the shelf labels 50 iscorrect.

FIG. 7 is a diagram illustrating a target image representing a casewhere the positional relationship is incorrect. In FIG. 7 , the shelflabel 50 is not located under the product group 40-1. Thus, thedetermination unit 2060 determines that there is an error in therelationship between the position of product groups 40 and the positionof shelf labels 50.

Here, there would be a case in which the shelf label is positioned in aboundary between two product groups, and the same shelf label ispositioned under the two product groups. FIG. 8 is a diagramillustrating a target image representing a case where a shelf label ispositioned in a boundary between two product groups. In FIG. 8 , theshelf label 50 is positioned in a boundary between two product groups40-1 and 40-2. The determination unit 2060 needs to determine underwhich product group the shelf label is located.

In such a case, the determination result performed by the determinationunit 2060 could be different based on under which product group 40 theshelf label 50 is determined to be located. If it is determined that theshelf label 50 is located under the product group 40-1, the shelf label50 is attached under each product group 40. Thus, the determination unit2060 determines that the relationship between the position of productgroups 40 and the position of the shelf labels 50 is correct. On theother hand, if it is determined that the shelf label 50 is located underthe product group 40-2, there is no shelf label 50 under the productgroup 40-1. Thus, the determination unit 2060 determines that there isan error in the relationship between the position of product groups 40and the position of shelf labels 50.

For example, when there is the shelf label 50 in the boundary betweentwo product groups 40 as stated above, the determination unit 2060determines that the shelf label 50 is located under the closer productgroup 40. The determination method will be described in detail withreference to FIG. 9 . FIG. 9 is a diagram illustrated by zooming an areanear the boundary between the product group and the product group 40-2of FIG. 8 . In order to easily view the diagram, the described contentof the shelf label 50-1 will not be repeated. In FIG. 9 , a horizontaldistance between the right end of the product group 40-1 and the centerof the shelf label 50-1 is d1. In FIG. 9 , a horizontal distance betweenthe left end of the product group 40-2 and the center of the shelf label50-1 is d2. Here, since d2 is less than d1, the product group 40-2 islocated closer to the shelf label 50-1 than the product group 40-1 is.Thus, the determination unit 2060 determines that the shelf label islocated under the product group 40-2.

As a result, in the example of FIG. 8 , there is no shelf label 50 underthe product group 40-1. Thus, the determination unit 2060 determinesthat there is an error in the relationship between the position ofproduct groups 40 and the position of shelf labels 50.

Note that, in a case where a plurality of rows of the product shelvingis pictured on the target image, the determination unit 2060 may takeinto consideration which product group 40 and which shelf label 50 areincluded in the same row. Specifically, the determination unit 2060limits the target of the determination of whether or not there is anerror in the positional relationship or the number relationship to theproducts and the shelf labels included in the same row.

Although it has been described in FIG. 4 that the product group 40corresponding to the shelf label 50 is placed above the shelf label 50,an exhibition method of placing the product group 40 corresponding tothe shelf label under this shelf label 50 is also used. That is, acertain row of the product shelving becomes 1) an area including theshelf plate 30 and the product group 40 placed on the shelf plate 30 or2) an area including the shelf plate 30 and the product group 40 placedunder the shelf plate 30. Whether the determination unit 2060recognizes 1) or 2) as the row is set in advance. For example,information for determining whether the determination unit recognizes 1)or 2) as the row (information indicating whether the shelf label 50 isattached above or under the product group 40) is stored in advance inthe storage unit provided inside or outside the information processingapparatus 2000. The product recognition unit 2020 determines an arearecognized as one row by referring to this information. The productrecognition unit 2020 determines the relationship between the productgroup 40 and the shelf label 50 included in the same row.

The determination unit 2060 may determine that there is an error in therelationship between the position of the product group 40 and theposition of the shelf label 50 in a case where there is no shelf label50 within a predetermined distance range from a certain product group40. In this case, it is assumed that the predetermined distance isstored in the storage unit outside or inside the information processingapparatus 2000. The determination unit 2060 may take both the directionand the distance into consideration. Specifically, the determinationunit 2060 determines that there is an error in the positionalrelationship between the product group 40 and the shelf label 50 in acase where there is no shelf label 50 that satisfies the condition inwhich “a certain product group 40 is located in a predetermineddirection with respect to the product group 40 and a distance from theproduct group 40 is equal to or less than a predetermined distance”.

<Determination Method of Focusing on Both Number Relationship andPositional Relationship>

The determination unit 2060 may determine an error in the relationshipbetween the product group 40 and the shelf label 50 by focusing on boththe number relationship and the positional relationship. Thedetermination method will be described in detail with reference to FIG.5 . In the situation of FIG. 5 , if the determination unit 2060determines an error in the relationship between the position of theproduct group 40 and the position of the shelf label 50, it isdetermined that there is an error in the positional relationship. Theshelf label 50 is not located under the product group 40-3. However, insome points of view, the situation of FIG. 5 may mean that “there is noerror in the positional relationship and there is an error in the numberrelationship”.

Thus, the determination unit 2060 may determine whether or not there isan error in the relationship between the position of the product group40 and the position of the shelf label 50, only when the relationshipbetween the number of product groups 40 and the number of shelf labels50 is correct. A flowchart for illustrating a flow of processes isillustrated in FIG. 10 .

The information processing apparatus 2000 acquires the target image(S202). The product recognition unit 2020 recognizes the number orpositions of product groups 40 from the target image (S204). The shelflabel recognition unit 2040 recognizes the number or positions of shelflabels 50 from the target image (S206). The determination unit 2060compares the number of product groups 40 with the number of shelf labels50 (S208).

In a case where the number of product groups 40 and the number of shelflabels 50 are not equal to each other (S210: NO), the process of FIG. 10proceeds to step S212. In step S212, the determination unit 2060determines that there is an error in the number relationship between theproduct groups 40 and the shelf labels 50.

On the other hand, in a case where the number of product groups 40 andthe number of shelf labels 50 are equal to each other (S210: YES), theprocess of FIG. 10 proceeds to step S214. In step S214, thedetermination unit 2060 compares the position of the product groups 40with the position of the shelf labels 50. In a case where there is aproduct group 40 that has no corresponding shelf label 50 (S216: YES),the process of FIG. 10 proceeds to step S218. In step S218, thedetermination unit 2060 determines that there is an error in thepositional relationship between the product groups 40 and the shelflabels 50. On the other hand, in a case where there is no product group40 that has no corresponding shelf label 50 (S216: NO), the process ofFIG. 10 is ended.

The determination is performed using both the number relationship andthe positional relationship as stated above, and the determinationresult performed by the determination unit 2060 becomes more accurate.

<Timing when Information Processing Apparatus 2000 Performs Processes>

There are various timings when the information processing apparatus 2000performs the processes. For example, the information processingapparatus 2000 periodically performs the processes. In this case, a dateand time or a cycle in which the process is performed is set in advancefor the information processing apparatus 2000. For example, theinformation processing apparatus 2000 may perform the process when theoperation by the salesperson is received. The information processingapparatus 2000 may perform the process when the target image isgenerated or when the target image is acquired.

Exemplary Embodiment 2

FIG. 11 is a block diagram illustrating an information processingapparatus 2000 according to Exemplary Embodiment 2. In FIG. 11 , arrowsrepresent information flows. In FIG. 11 , each block does not representa configuration of a hardware unit but a configuration of a functionalunit.

The information processing apparatus 2000 of Exemplary Embodiment 2includes a notification unit 2080. The notification unit 2080 performsthe notification or the displaying in a case where it is determined thatthere is an error by the determination unit 2060. By performing thenotification or the displaying as stated above, the salesperson caneasily recognize the determination result performed by the determinationunit 2060.

<Flow of Processes>

FIG. 12 is a flowchart illustrating a flow of processes performed by theinformation processing apparatus 2000 of Exemplary Embodiment 2. Here,steps S102 to S108 are the same as those of the flowchart of FIG. 2 ,and thus, the description thereof will not be repeated. The process ofFIG. 12 is branched at step S302. Specifically, in a case where it isdetermined that there is no error in the number or positionalrelationship between the product groups 40 and the shelf labels 50 instep S108 (step S302: NO), the process of FIG. 12 is ended. On the otherhand, in a case where it is determined that there is the error in thenumber or positional relationship between the product groups 40 and theshelf labels 50 in step S108 (step S302: YES), the process of FIG. 12proceeds to step S304. In step S304, the notification unit 2080 performsthe notification or displaying indicating that there is the error in thenumber or positional relationship between the product groups 40 and theshelf labels 50.

<About Target Terminal of Notification or Display>

The salesperson present in a store into which the information processingapparatus 2000 is introduced may directly use the information processingapparatus 2000 or may use another terminal (hereinafter, referred to asa user terminal) connected to the information processing apparatus 2000in a communicatable fashion. For example, the user terminal is aportable terminal carried by each salesperson.

For example, in a case where the salesperson directly uses theinformation processing apparatus 2000, when the determination unit 2060determines that there is the error in the number or positionalrelationship between the product groups 40 and the shelf labels 50, thenotification unit 2080 displays the notification indicating that thereis the error, on the display screen connected to the informationprocessing apparatus 2000. Meanwhile, in a case where the salespersonuses the user terminal, when it is determined that there is the error inthe number or positional relationship between the product groups 40 andthe shelf labels 50, the information processing apparatus 2000 notifiesthe user terminal of the determination result performed by thedetermination unit 2060. The user terminal that receives thisnotification displays the determination result on the display screenincluded in the user terminal. Here, the notification unit 2080 mayperform both the displaying on the information processing apparatus 2000and the notification to the user terminal.

Here, the process performed by the information processing apparatus 2000or the user terminal in a case where it is determined that there is theerror in the number or positional relationship between the productgroups 40 and the shelf labels 50 is not limited to the output to thedisplay screen. For example, the information processing apparatus 2000or the user terminal may output voice for reporting that it isdetermined that there is the error, or may perform a process of turningon the light.

<Notification Method of Error Content>

The notification unit 2080 may notify of or display the “notificationindicating that there is the error” or may notify of or display the“notification indicating the kind of the error”. For example, thenotification unit 2080 notifies of information for determining alocation (product shelving or row) in which it is determined that thereis the error. Accordingly, the salesperson can easily recognize alocation in which it is necessary to add or remove the products or theshelf labels. Note that, the notification unit 2080 performs thedisplaying on the display screen by the same method as variousnotification methods to be described below.

<<Notification Method of Target Location>>

For example, the notification unit 2080 performs notification includingthe target image on which a target location of notification (location inwhich the error is found in the relationship between the product groups40 and the shelf labels 50) is pictured. Here, in a case where aplurality of locations (product shelving) in which the products areexhibited is pictured on the target image, the notification unit 2080may attach the image on which the target location is highlighted or mayattach the image on which only the target location is cropped.

For example, the notification unit 2080 may determine the location asthe notification target, and may notify information related to thislocation. For example, this information is an ID of the product shelvingor a position within the store.

In a case where the ID of the product shelving is displayed on theproduct shelving, the notification unit 2080 recognizes the ID of theproduct shelving as the notification target that is pictured on thetarget image. The notification unit 2080 notifies of the recognized IDof the product shelving. The notification unit 2080 may determine theposition of the product shelving within the store by using the ID of theproduct shelving, and may notify of this position. In this case, thestorage unit inside or outside the information processing apparatus 2000stores information that associates the ID of the product shelving withthe position of the product shelving within the store that has this ID(hereinafter, product shelving information).

The notification unit 2080 may determine the position of the productshelving within the store as the notification target and may notify ofthis position by using information related to the arrangement of theproduct shelving (store map) and information related to the imagingrange of the camera with which the target image is imaged. Here, theinformation related to the imaging range of the camera that images thetarget image may be acquired from the camera, or may be acquired fromthe storage unit that stores the information related to the camera.Here, the notification unit 2080 may determine the ID of the productshelving from the position of the product shelving within the store asthe notification target within the store and may notify of this ID byfurther using the aforementioned product shelving information.

The notification unit 2080 may notify of the information related to therow (row in which the error is found in the relationship between theproduct groups 40 and the shelf labels 50) as the notification target bythe same method as the method of notifying of the information related tothe aforementioned product shelving.

<Notification Method of Product Group 40 being Notification Target>

For example, in a case where there is no shelf label above or under theproduct group 40, the notification unit 2080 preferably notifies thatthere is no shelf label 50 corresponding to the product group 40. Here,there are various methods of notifying of the product group 40 as thisnotification target. For example, the notification unit 2080 may performthe notification including the image on which the product group 40 ofthe notification target is highlighted or the image on which only theproduct group 40 as the notification target is cropped from the targetimage.

For example, the notification unit 2080 determines the product group 40as the notification target by using the aforementioned productinformation (information in which the product is associated with thefeature value indicating the appearance feature of the product). Here,the feature value of the product of each product group 40 is alreadycalculated by the product recognition unit 2020. Thus, the notificationunit 2080 determines the product corresponding to the feature value ofthe product of the product group 40 as the notification target by usingthe product information. For example, the notification unit 2080notifies of the ID of the determined product group 40.

An ID for identifying the product may be assigned to the product. ThisID may be represented by the character string such as alphabetical oralphanumeric characters, or may be represented by an image such asbarcode or two-dimensional code. This ID is identified by the productrecognition unit 2020. In this case, the notification unit 2080 notifiesof the ID of the product of the notification target.

<<Determination Method of Shelf Label 50 of Notification Target>>

For example, in a case where there is no product group above or underthe shelf label 50, the notification unit 2080 preferably notifies thatthere is no product corresponding to the shelf label 50. For example,the notification unit 2080 may perform the notification including theimage on which the shelf label 50 of the notification target ishighlighted or the image on which only the shelf label 50 of thenotification target is cropped from the target image.

The ID may be assigned to the shelf label by the same method as theaforementioned method for assigning the ID to the product. This ID isidentified by the shelf label recognition unit 2040. In this case, thenotification unit 2080 notifies of the ID of the shelf label of thenotification target.

<Notification Indicating that there is No Error>

In a case where it is determined that there is no error in therelationship between the product groups 40 and the shelf labels 50, thenotification unit 2080 may notify or display that there is no error. Ina case where it is determined that there is no error, the notificationunit 2080 may notify of or display the information related to thelocation, the product group or the shelf label in which it is determinedthat there is no error.

Although the exemplary embodiments of the present invention have beendescribed with reference to the drawings, these exemplary embodimentsare examples of the present invention, and the combinations of theexemplary embodiments and various configurations other than theexemplary embodiments may be adopted.

Hereinafter, reference examples are appended.

-   -   1. An information processing apparatus including:    -   a product recognition unit recognizing a number of types of        products or positions of each type of products from an image on        which the products and shelf labels are imaged;    -   a shelf label recognition unit recognizing a number or positions        of shelf labels indicating information of the products from the        image; and    -   a determination unit determining whether or not there is an        error in a relationship between the recognized number of types        of products and the recognized number of shelf labels or in a        relationship between the recognized positions of each type of        products and the recognized positions of the shelf labels.    -   2. The information processing apparatus according to 1,    -   wherein, in a case where the recognized number of types of        products and the recognized number of shelf labels are not equal        to each other, the determination unit determines that there is        an error in the relationship between the recognized number of        types of products and the recognized number of shelf labels.    -   3. The information processing apparatus according to 1 or 2,    -   wherein, in a case where the shelf label is not attached for a        predetermined type of products in a predetermined direction, the        determination unit determines that there is an error in the        relationship between the recognized positions of each type of        products and the recognized positions of the shelf labels.    -   4. The information processing apparatus according to any one of        1 to 3,    -   wherein, in a case where the shelf label is not attached within        a predetermined distance range from a predetermined type of        products, the determination unit determines that there is an        error in the relationship between the recognized positions of        each type of products and the recognized positions of the shelf        labels.    -   5. The information processing apparatus according to any one of        1 to 4,    -   wherein, in a case where it is determined that there is an error        in the relationship between the recognized number of types of        products and the recognized number of shelf labels, the        determination unit determines whether or not there is an error        in the relationship between the recognized positions of each        type of products and the recognized positions of the shelf        labels.    -   6. The information processing apparatus according to any one of        1 to 5, further including:    -   a notification unit performing notification or displaying in a        case where the determination unit determines that there is an        error in the relationship between the recognized number of types        of products and the number of shelf labels or in the        relationship between the recognized positions of each type of        products and the recognized positions of the shelf labels.    -   7. A control method performed by a computer, the method        including:    -   a product recognition step of recognizing a number of types of        products or positions of each type of products from an image on        which the products and shelf labels are imaged;    -   a shelf label recognition step of recognizing a number or        positions of shelf labels indicating information of each product        from the image; and    -   a determination step of determining whether or not there is an        error in a relationship between the recognized number of types        of products and the recognized number of shelf labels or in a        relationship between the recognized positions of each type of        products and the recognized positions of the shelf labels.    -   8. The control method according to 7,    -   wherein, in a case where the recognized number of types of        products and the recognized number of shelf labels are not equal        to each other, it is determined that there is an error in the        relationship between the recognized number of types of products        and the recognized number of shelf labels in the determination        step.    -   9. The control method according to 7 or 8,    -   wherein, in a case where the shelf label is not attached for a        predetermined type of products in a predetermined direction, it        is determined that there is an error in the relationship between        the recognized positions of each type of products and the        recognized positions of the shelf labels in the determination        step.    -   10. The control method according to any one of 7 to 9,    -   wherein, in a case where the shelf label is not attached within        a predetermined distance range from a predetermined type of        products, it is determined that there is an error in the        relationship between the recognized positions of each type of        products and the recognized positions of the shelf labels in the        determination step.    -   11. The control method according to any one of 7 to 10,    -   wherein, in a case where it is determined that there is an error        in the relationship between the recognized number of types of        products and the recognized number of shelf labels, it is        determined whether or not there is an error in the relationship        between the recognized positions of each type of products and        the recognized positions of the shelf labels in the        determination step.    -   12. The control method according to any one of 7 to 11, further        including:    -   a notification step of performing notification or displaying in        a case where it is determined that there is an error in the        relationship between the recognized number of types of products        and the number of shelf labels or in the relationship between        the recognized positions of each type of products and the        recognized positions of the shelf labels in the determination        step.    -   13. A program causing a computer to operate as the information        processing apparatus according to any one of claims 1 to 6.

The present application claims priority based on Japanese PatentApplication No. 2014-201363 filed on Sep. 30, 2014, and the entirecontent thereof is incorporated by reference in the present application.

1. An information processing apparatus comprising: at least one memoryconfigured to store instructions; and at least one processor configuredto execute the instructions to: recognize a position of at least oneproduct from an image on which products and a shelf label are imaged;recognize a position of the shelf label indicating information of aproduct from the image; detect an error in a relationship between therecognized position of the product and the recognized position of theshelf label; and output the image on which a location including theposition of the product related to the detected error is highlighted. 2.The information processing apparatus according to claim 1, wherein theprocessor is further configured to execute the instructions to: when theimage is generated, or when the image is acquired, recognize theposition of at least one product and the position of the shelf label. 3.The information processing apparatus according to claim 1, wherein theinformation processing apparatus is a portable terminal.
 4. Theinformation processing apparatus according to claim 3, wherein theprocessor is further configured to execute the instructions to: outputthe image on a display screen of the portable terminal.
 5. Theinformation processing apparatus according to claim 3, wherein theprocessor is further configured to execute the instructions to: generatethe image by a capture unit of the portable terminal.
 6. The informationprocessing apparatus according to claim 1, wherein the processor isfurther configured to execute the instructions to: perform notificationor displaying that there is an error, when there the error in therelationship between the recognized position of the product and therecognized position of the shelf label is detected.
 7. The informationprocessing apparatus according to claim 1, wherein the processor isfurther configured to execute the instructions to: perform notificationor displaying that there is no error, when there the error in therelationship between the recognized position of the product and therecognized position of the shelf label is not detected.
 8. A controlmethod performed by a computer, the method comprising: recognizing aposition of at least one product from an image on which products and ashelf label are imaged; recognizing a position of the shelf labelindicating information of a product from the image; detecting an errorin a relationship between the recognized position of the product and therecognized position of the shelf label; and outputting the image onwhich a location including the position of the product related to thedetected error is highlighted.
 9. The control method according to claim8, further comprising: by the computer, when the image is generated, orwhen the image is acquired, recognizing the position of at least oneproduct and the position of the shelf label.
 10. The control methodaccording to claim 8, wherein the computer is a portable terminal. 11.The control method according to claim 10, further comprising: by thecomputer, outputting the image on a display screen of the portableterminal.
 12. The control method according to claim 10, furthercomprising: by the computer, generating the image by a capture unit ofthe portable terminal.
 13. The control method according to claim 8,further comprising: by the computer, performing notification ordisplaying that there is an error, when there the error in therelationship between the recognized position of the product and therecognized position of the shelf label is detected.
 14. The controlmethod according to claim 8, further comprising: by the computer,performing notification or displaying that there is no error, when therethe error in the relationship between the recognized position of theproduct and the recognized position of the shelf label is not detected.15. A non-transitory computer-readable storage medium storing a programcausing a computer to: recognize a position of at least one product froman image on which products and a shelf label are imaged; recognize aposition of the shelf label indicating information of a product from theimage; detect an error in a relationship between the recognized positionof the product and the recognized position of the shelf label; andoutput the image on which a location including the position of theproduct related to the detected error is highlighted.
 16. Thenon-transitory computer-readable storage medium according to claim 15,wherein the program further causes the computer to: when the image isgenerated, or when the image is acquired, recognize the position of atleast one product and the position of the shelf label.
 17. Thenon-transitory computer-readable storage medium according to claim 15,wherein the computer is a portable terminal.
 18. The non-transitorycomputer-readable storage medium according to claim 17, wherein theprogram further causes the computer to: output the image on a displayscreen of the portable terminal.
 19. The non-transitorycomputer-readable storage medium according to claim 17, wherein theprogram further causes the computer to: generate the image by a captureunit of the portable terminal.
 20. The non-transitory computer-readablestorage medium according to claim 15, wherein the program further causesthe computer to: perform notification or displaying that there is anerror, when there the error in the relationship between the recognizedposition of the product and the recognized position of the shelf labelis detected.